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(54) Method of applying manipulations to a color digital image 

(57) A method for modifying a digital image having 
color values with an extended color gamut comprising 
the steps of representing the extended color gamut dig- 
ital image with a limited color gamut digital image and 
one or more associated residual image(s) representing 
a difference between the extended color gamut digital 
image and the limited color gamut digital image; speci- 
fying one or more desired image modification(s) to be 
applied to the extended color gamut digital image; and 
modifying the limited color gamut digital image and the 
residual image(s) to form a modified limited color gamut 
digital image and one or more modified residual 
image(s) in response to the specified desired image 
modification(s). 



eXTENOED COUOR 
GAMUT DXSUAL imCE 



AOJUSr DOLOR VALUES 
TO F(T WfTHIN UMTTB) 
CaOR GAMUT 



UMITEO COLOR GAMUT 
OiGnAi /MA6E 



COMPUTE 
REStOVAL IMAGE 



REPRESENT IMAGE IN 
STORAGE COlOfl SPACE 



STORAGE SPACE X 
DIGITAL IMAGES 



RESJDUAL IMAGE 



CNJ 
< 



STORE fN mm. FILE 



F/G. 2 



o> 
o 

Q. 
LJJ 



Printed by Xerox {UK} E3usiness Services 
^16.7.'3,6 



1 



EP0 991 019 A2 



2 



Description 

[0001] The present invention reiates to the field of dig- 
ital imaging, and more particularly to manipulating an 
extended color gamut digital image. s 
[0002] In digital imaging systems, there are many 
ways to represent images in digital form. Not only are 
there many different formats of digital files, but there are 
also a large variety of different color spaces and color 
encodings that can be used to specify the color of digital io 
images. 

[0003] In some cases, the color encoding can be In 
terms of a so-called device independent color space, 
such as the well-known CIELAB color space. In recent 
years this color space has been used extensively to is 
specify the color of digital images in color-managed dig- 
ital imaging systems. In some cases, the image can 
actually be stored in the CIELAB color space. More 
commonly, the color space can be used to connect 
device profiles, which can be used to describe the color 20 
characteristics of various color imaging devices such as 
scanners, printers, and CRT video displays. The 
KODAK Photo YCC Color Interchange Space is another 
example of a device independent color space that can 
be used to encode digital images. 25 
[0004] In other cases, the color-encoding can be in 
terms of a device dependent color space. Video RGB 
color spaces and CMYK color spaces are examples of 
this type. When a color image is encoded in a device 
dependent color space, it will have the desired color 30 
appearance when it is displayed on the particular output 
device associated with that color space. The advantage 
of a device dependent color space is that the image is 
ready to be displayed or printed on the target device. 
However the disadvantage is that the Image will neces- 35 
sarily be limited to the color gamut of the target device. 
The color gamut of an imaging device refers to the 
range of colors and luminance values that can be pro- 
duced by the device. Therefore, if the target device has 
a limited dynamic range, or is incapable of reproducing 40 
certain saturated colors, then it is not possible to 
encode color values outside of the range of colors that 
can be produced on the device. 
[0005] One type of device dependent color space that 
has become quite widespread for use as a storage and 45 
manipulation color space for digital images is the video 
RGB color space. In reality, there are many different 
video RGB color spaces due to the fact that there are 
many different types of video RGB displays. As a result, 
a particular set of video RGB color values will corre- so 
spond to one color on one video display and to another 
color on another video display. Therefore, video RGB 
has historically been a somewhat ambiguous color rep- 
resentation due to the fact that the color values can not 
be properly interpreted unless the characteristics of the 55 
target video display were known. Nonetheless, video 
RGB color spaces have become the defacto standard in 
many applications because the creation, display and 



editing of images on video displays are central steps in 
many digital imaging systems. 

[0006] Recently, there have been efforts to standard- 
ize a particular video RGB color space in order to 
remove the ambiguity in the interpretation of the color 
values. (See the proposed lEC TC100 sRGB Draft 
Standard). One such proposed standard color space is 
known as "sRGB." This color space specifies a particu- 
lar set of red, green, and blue primaries, a particular 
whitepoint, and a particular non-linear code value to 
light intensity relationship. Together, these tightly define 
the overall relationship between the digital code values 
and the corresponding device Independent color values. 
[0007] Although the use of a standard video RGB 
color space eliminates much of the ambiguity usually 
associated with video RGB color spaces, it does noth- 
ing to address the fact that this color space has a limited 
color gamut relative to other output devices. Addition- 
ally, any output device will have a limited color gamut 
relative to that of an original scene. For example, a 
scene can have a luminance dynamic range of 1000:1 
or more, whereas a typical video display or reflection 
print will have a dynamic range on the order of 100:1. 
Certain image capture devices, such as photographic 
negative film, can actually record dynamic ranges as 
large as 8000:1. Even though this Is larger than the 
luminance dynamic range associated with most scenes, 
the extra dynamic range is often useful to provide allow- 
ance for exposure errors, light source variations, etc. 
[0008] In order to encode images from various 
sources in a video RGB representation, it is necessary 
to discard information that is outside the color gamut of 
the video RGB color space. In some cases, such as 
when it is desired to encode the appearance of colors In 
an original scene or the colors captured by a photo- 
graphic negative, a great deal of information will typi- 
cally need to be discarded due to the large disparity in 
the dynamic ranges. For the case where it is desired to 
scan a reflection print and store it in a video RGB color 
space, it is still necessary to discard a substantial 
amount of information due to the mismatch in the color 
gamuts, even though the luminance dynamic ranges 
can be quite similar. 

[0009] For example, FIG. 1 shows a comparison of a 
typical Video RGB Color Gamut 1 0 and a typical Reflec- 
tion Print Color Gamut 12. In this case, a*-b* cross-sec- 
tions of the color gamuts are shown in the CIELAB 
space at an L* of 65. The colors that are inside the 
boundary are within the gamuts of the respective 
devices, while those that are outside the boundary can- 
not be reproduced, and are therefore referred to as "out- 
of-gamut" colors. It can be seen that there is a large set 
of color values with a b* value larger than 60 that can be 
produced on the printer, but are outside the color gamut 
of the video display. As a result, if the reflection print 
were scanned and stored in a video RGB color space, it 
would not be possible to encode this color information. 
[0010] The mismatch between the video RGB color 
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gamut and the color gamuts of other output devices and 
image sources represents a serious limitation on the 
usefulness of the video RGB color space. However, in 
many cases, the convenience of storing the image in a 
color space that is ready for direct display on a compu- 5 
ter video CRT has been the over-riding factor in the 
determination of the preferred color space. This has 
come at the expense of applications that can utilize the 
extended color gamut information that can have existed 
in an input image. One of the most serious limitations of 10 
this approach involves cases where it is desired to mod- 
ify the digital image. In this case, it is frequently benefi- 
cial to use the information that must be clipped off when 
representing the image in a video RGB color gamut to 
provide a better quality modified image. For example, if /s 
an image needs to be darkened due to the fact that the 
original image was over-exposed, the video RGB image 
will generally not retain the highlight detail information 
that could be used to darken the image. 
[0011] It is an object of the present invention to over- 20 
come the limitations of the prior art by providing for the 
modification of digital images stored in a color space 
having a limited color gamut by retaining extended color 
gamut information. 

[0012] This object is achieved in a method for modify- 25 
ing a digital image having color values with an extended 
color gamut comprising the steps of: 

a) representing the extended color gamut digital 
image with a limited color gamut digital image and 30 
one or more associated residual image(s) repre- 
senting a difference between the extended color 
gamut digital image and the limited color gamut dig- 
ital image; 

b) specifying one or more desired image moditica- 35 
tion(s) to be applied to the extended color gamut 
digital image; and 

c) modifying the limited color gamut digital image 
and the residual image(s) to form a modified limited 
color gamut digital image and one or more modified 40 
residual image(s) in response to the specified 
desired image modification(s). 

[0013] The present invention has the advantage that 
a digital image can be stored in a color space conven- 45 
lent for a particular application while overcoming the 
color gamut limitation associated with that color space. 
[0014] It has the additional advantage that the use of 
the extended color gamut information is optional. As a 
result, the benefits of the extended color gamut informa- so 
tion can be gained by applications that are able to make 
use of it, without introducing an image quality or compu- 
tation penalty for applications that do not require the 
optional information or that are not able to make use of 
it. 55 
[0015] It has the additional advantage that the 
extended color gamut information can be used to mod- 
ify the image so as to provide a larger amount of adjusti- 



bility in the image modification process. 
[0016] The image can be stored, for example, in a 
video RGB color space that is well-adapted for fast and 
convenient display on a computer system without com- 
promising the potential quality of the image. This pro- 
vides a particular advantage during the process of 
applying many types of modifications to the digital 
image. 

FIG. 1 is graph comparing the color gamuts of a 
typical video display, and a typical reflection print; 
FIG. 2 is a flowchart showing a process for making 
a limited gamut digital image in accordance with the 
present invention; and 

FIG. 3 is a flowchart showing a second process for 
making a limited gamut digital image in accordance 
with the present invention; 
FIG. 4 is a flowchart showing the reconstruction of 
a extended gamut digital image from the limited dig- 
ital image of FIG. 2; 

FIG. 5 is a flowchart showing the modification of the 
limited color gamut digital image and the residual 
image(s) in accordance with the present invention; 
FIG. 6 is a flowchart showing the modification of the 
extended color gamut digital image in accordance 
with the present invention; 

FIG. 7 is a flowchart showing the storing of informa- 
tion about desirable image modification(s) in 
accordance with the present invention. 

[0017] One method for storing a digital image in 
accordance with the present invention is shown in 
FIG. 2. An extended color gamut digital image 20 has 
color values that are outside the limited color gamut of a 
storage color space. An adjust color values step 21 is 
used to limit the color values to those that will fit within 
the limited color gamut of the storage color space to 
form a limited color gamut digital image 22. Next, a rep- 
resent image in storage color space step 23 is used to 
produce a storage space digital image 24. A compute 
residual image{s) step 25 is used to determine one or 
more residual image(s) 26 representing the difference 
between the extended color gamut digital image and the 
limited color gamut digital image. The storage space 
digital image 24 and the residual image(s) 26 are then 
stored in a digital file 28 using a digital file storage step 
27. 

[001 8] A key aspect of the present invention is the cre- 
ation and use of the residual image(s) 26 representing 
the difference between the extended color gamut digital 
image and the limited color gamut digital image. Other 
prior art systems include the computation of a residual 
image, but none involve computing a difference 
between an extended color gamut digital image and a 
limited color gamut digital image. Nishihara et al. (U.S. 
Patent 4,903,317) describe the computation of a resid- 
ual image determined from the difference between an 
original image, and an image that has been com- 
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pressed using a lossy data compression technique and 
subsequently decompressed. The residual image rep- 
resents the compression artifacts that are introduced 
during the compression/decompression process. Golin 
(U.S. Patent 5,122,873) also describes a method for 5 
encoding images using a residual image. In this case, 
the residual image relates to the difference between 
images of different spatial resolution. It is also known 
that a residual image can be computed between a high- 
precision digital image, and a low-precision digital w 
image. In each of these cases, the images being differ- 
enced have identical color gamuts and color spaces. As 
a result, none of these prior art configurations would 
support the storage of extended color gamut informa- 
tion as required in the present invention. /5 
[0019] Each of the steps in FIG. 2 will now be dis- 
cussed in more detail. The extended color gamut digital 
image 20 can take many different forms. For example, 
the image can be a scanned photographic print, a 
scanned photographic negative, a scanned photo- 20 
graphic transparency, or an image from a digital cam- 
era, etc. Depending on the source of the image, as well 
as any image processing that has been applied to the 
image, the image can have very different color gamuts 
and color representations. Images from scanned photo- 25 
graphic negatives and digital cameras can contain 
scene information having a much larger luminance 
dynamic range than can be encoded in many storage 
color spaces. In this case, luminance dynamic range is 
simply one aspect of color gamut related to the range of 30 
luminance values that can be represented. 
[0020] The color gamut of an imaging system is the 
range of colors that can be represented or produced. 
Since color is fundamentally a three-dimensional phe- 
nomenon, color gamuts can be viewed as a three- 35 
dimensional volume. Color values that are within the 
volume are said to be "in-gamut," whereas colors that 
are outside the volume are said to be "out-of-gamut." 
One aspect of the color gamut is the luminance dynamic 
range of the system. This is the range of relative lumi- 40 
nance values that can be encoded by the system from 
the whitest white to the blackest black. Another aspect 
of the color gamut is the range of chroma values that 
can be represented from a neutral out to a saturated 
color. The range of chroma values that are in-gamut will 45 
generally be a function of hue and lightness. Generally, 
the highest chroma colors can be produced near the 
hue and lightness of the primary and secondary colors 
of a given imaging device or color space {usually red, 
green, blue, cyan, magenta and yellow). so 
[0021] If the image were a scanned photographic 
print, the color gamut of the image would generally be 
the cplor gamut of the original photographic print 
medium. Likewise if the image were captured by a dig- 
ital camera, the color gamut of the image would gener- 55 
ally be that of an original scene, although it can be 
limited by the dynamic range of the camera sensor and 
by lens flare. The color space that the image is repre- 



6 

sented in is somewhat independent of the color gamut 
of the original image. For example, the color values for a 
scanned photograph can be represented as raw scan- 
ner code values, or they can be given by device inde- 
pendent color values according to a color space such as 
the CIELAB color space. Alternatively, the color values 
can be expressed in some other color space. 
[0022] For many applications, it is convenient to store, 
display and manipulate the digital image in a particular 
storage color space that is well-suited for the work flow 
associated with that application. Frequently the storage 
color space that is chosen will be a device dependent 
color space associated with a common output device or 
medium used by the system. In many cases, video RGB 
color spaces are used because they can be displayed or 
previewed directly on a computer video display without 
any further processing. Additionally, many software 
applications that are available to manipulate images on 
a computer are designed to work with images in a video 
RGB color space. The color gamut of the storage color 
space will often be smaller than, or at least different 
than, the color gamut of the extended color gamut digital 
image 20. As a result, there generally will be colors in 
the extended color gamut digital image 20 that can not 
be represented in the storage color space. For example, 
consider the case where the extended color gamut dig- 
ital image 20 is a scanned photographic print. There are 
many colors within the color gamut of the reflection print 
that are outside the color gamut of the video RGB color 
space. This can clearly be seen in FIG. 1 which shows 
cross- sections through a typical Video RGB Color 
Gamut 10, and a typical Reflection Print Color Gamut 
12. 

[0023] Therefore, information must be discarded in 
order to store the reflection print color values in a video 
RGB color space, or any other limited color gamut stor- 
age space. In prior art methods, the information that is 
discarded is lost permanently andean not be recovered. 
In the present invention, the information that is lost will 
be stored in one or more residual image(s). 
[0024] The adjust color values step 21 is used to 
adjust the color values of the extended color gamut dig- 
ital image to fit within the limited color gamut of the stor- 
age space, forming a limited color gamut digital image 
22. In this step, information must be discarded when 
color values that are outside the limited color gamut are 
mapped to color values within the limited color gamut. In 
some cases, the color values for the out-of-gamut colors 
are simply "clipped," i.e., they are mapped to color val- 
ues the surface of the limited color gamut. In other 
cases, more sophisticated gamut mapping methods can 
be used to compress the extended color gamut into the 
limited color gamut without introducing a hard clipping 
function. For example, the chroma of the input color val- 
ues can be scaled so that the most saturated colors in 
the extended color gamut are mapped to the most satu- 
rated colors in the limited color gamut. Alternatively, a 
gamut mapping method can be used that is designed to 
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preserve color appearance as closely as possible. 
Regardless of what gamut mapping technique is used, 
there will necessarily be a loss of information and a dis- 
tortion of the color characteristics of the image. 
[0025] In many cases, the extended color gamut will 
contain color values that have higher chroma values 
than can be represented in the limited color gamut. In 
some cases, the extended bDlor gamut can also have a 
larger luminance dynamic range than can be repre- 
sented in the limited color gamut. In the case where it is 
necessary to reduce the luminance dynamic range of 
the image, one part in the implementation of the adjust 
color values step 21 is typically the application of a tone 
scale function. The tone scale function maps the input 
image intensities to output image intensities and might 
be applied to a luminance channel of the image, or alter- 
natively to each color channel of an RGB color repre- 
sentation. In some applications, the image being 
processed can actually be a monochrome image, e.g., a 
black-and-white image. In this case, the tonescale func- 
tion can be applied to the monochrome image lumi- 
nance values. 

[0026] For cases where the extended color gamut dig- 
ital image is a representation of the colors in an original 
scene, the adjust color values step 21 will typically 
involve determining reproduced color values that will 
produce desired aim colors on a target output device. 
For example, optimal color reproduction aims can be 
applied to determine desired video RGB aim colors for 
the original scene colors. The process of transforming 
the original scene color values into aim reproduced 
color values is sometimes referred to as "rendering" the 
image. 

[0027] Once the limited color gamut digital image 22 
has been determined, the next step is to represent it in 
the storage color space using the represent image in 
storage color space step 23. The output of this step is a 
storage space digital image 24. This step typically 
involves applying a device model, or a color space con- 
version, to determine the storage space color values 
that correspond to the adjusted color values of the lim- 
ited color gamut digital image 22. For example, if the 
adjusted color values were specified in terms of the 
CIELAB color space, a video display device model can 
be used to determine the corresponding video RGB val- 
ues that would be necessary to produce the specified 
adjusted color values. 

[0028] A compute residual image(s) step 25 is used to 
determine one or more residual image(s) 26 represent- 
ing the difference between the extended color gamut 
digital image 20 and the limited color gamut digital 
image 22. In its simplest form, a single residual image 
26 can be calculated by simply subtracting the adjusted 
color values of the limited color gamut digital image 22 
from the input color values of the extended color gamut 
digital image 20. The residual image would then be in 
terms of the color space used to represent those color 
values. Alternatively, the color values can be trans- 
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formed into some other space that would be useful for 
computing the residual image. For example, it might be 
desirable to compute the residual image in a color 
space that is well-suited for compressing the residual 

5 image or that is convenient for use in reconstructing the 
extended color gamut digital image. 
[0029] There are several reasons why it may be 
advantageous to store multiple residual images instead 
of just a single residual image. For example, it might be 

10 desirable to store residual errors associated with lumi- 
nance errors in one residual image, and residual errors 
associated with chrominance errors in additional resid- 
ual images. This would enable an application to make a 
choice about which types of residual errors it would use 

15 during the process of determining a reconstructed 
extended color gamut digital image. 
[0030] In another case, a set of multiple residual 
images can correspond to different subsets of extended 
dynamic range Image data. For example, a first residual 

20 image can extend the dynamic range of the digital 
image some fixed amount beyond the dynamic range 
associated with the limited color gamut digital image. A 
second residual image can then extend the dynamic 
range an additional increment beyond the extended 

25 dynamic range associated with the first residual image. 
In this way, an application using the extended color 
gamut digital image can use only the residual image(s) 
associated with the amount of extended dynamic range 
required by the application. 

30 [0031] Another reason that using multiple residual 
images is useful is for cases where the residual images 
are stored in tags in the digital file having a limited size. 
In this case, the residual image data can be broken into 
smaller pieces that would fit within the size limitations. 

35 For example, residual images can be determined for 
subsets of pixels in the extended color gamut digital 
image. In this way, the residual image data can be 
stored in a tiled fashion. 

[0032] Generally, the extended color gamut digital 
40 image and the limited color gamut digital image 22 
should be represented in the same color space before 
the residual image{s) are calculated so that the in- 
gamut colors will be given by zero residual errors. Since 
most images will only have a small fraction of color val- 
45 ues that are out of gamut, the residual image(s) will be 
dominated by zeros, and therefore will be highly com- 
pressible. 

[0033] For the remainder of this disclosure, the case 
of a single residual image will be described. However, it 

50 should be recognized that the method can be easily 
generalized to use a set of multiple residual images, 
[0034] For cases where the adjust color values step 
21 involves the application of a transform that modifies 
the color values for the colors within the limited color 

55 gamut as well as those outside the limited color gamut, 
the residual image is determined by directly computing 
the difference between the input color values of the 
extended color gamut digital Image and the adjusted 
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color values of the limited color gamut digital image 22 
would have a large number of non-zero values. This can 
be undesirable for cases where the residual image is to 
be compressed. The prior example where the extended 
color gamut digital image is a representation of the orig- 
inal scene, and the adjust color values step 21 includes 
rendering the color values to determine desirable color 
values for a target output device, will generally suffer 
from this problem. In this case, it may be desirable to 
apply a second rendering function to the extended color 
gamut digital image to determine a second set of ren- 
dered color values that are desirable for another output 
device having a larger color gamut than the first target 
output device. If the second rendering function were 
identical to the first rendering function throughout most 
of the color gamut, then a residual image computed by 
taking the difference between the first and second ren- 
dered images would again be largely dominated by zero 
differences. In one preferred embodiment of the present 
invention, the first rendering function can produce a ren- 
dered image that is optimized for a video display, and 
the second rendering function can produce a rendered 
image that is optimized for some hypothetical output 
device having an idealized large color gamut. 
[0035] Once the residual image 26 has been calcu- 
lated, it should be associated in some fashion with the 
storage space digital image 24. This can involve storing 
the residual image 26 in a memory buffer that is associ- 
ated with a second memory buffer used to store the 
storage space digital image 24. Alternatively, many 
applications will store the image data in a digital file 28 
on some sort of digital storage media such as a mag- 
netic disk, an optical disk, or a PCMCIA card using a 
digital file storage step 27. In this case, the storage 
space digital image 24 and the residual image 26 can 
be stored in two different files, or can be stored in the 
same digital image file. In many cases, the file format 
used to store the storage space digits! image 24 may 
support the use of private image tags. For example, the 
file formats TIFF, EXIF and FlashPlX all support tags of 
this sort. These tags are sometimes referred to as meta- 
data. In cases where file formats of this type are used, it 
will be convenient to store the residual image data in the 
form of a residual image tag. In this way, applications 
that do not know how to make use of the residual image 
tag will simply ignore it, and will therefore have access 
only to the storage space digital Image 24. Whereas 
applications that know how to use the residual image 
tag will be able to make use of it to reconstruct the 
extended color gamut digital image. Some tile formats 
place a limit on the size of tags, so compression of the 
residual image is important for these applications. 
[0036] A second method of storing a digital image in 
accordance with the present invention is shown in FIG. 
3. This embodiment is similar to that shown in FIG. 2 but 
differs in the way that the residual image is determined. 
In this second embodiment, the residual image is repre- 
sented relative to the code values of the storage color 
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space. FIG. 3 shows an extended color gamut digital 
image 30, having color values that are outside the lim- 
ited color gamut of a storage color space. An adjust 
color values step 31 is used to limit the color values to 

5 those that will fit within the limited color gamut of the 
storage color space. Next, a represent image in storage 
color space step 32 is used to produce a limited color 
gamut digital image 33. A represent image in extended 
storage color space step 34 is then applied to the origi- 

10 nal image, and a compute residual image step 35 is 
used to determine a residual image 36 representing the 
difference between the extended color gamut digital 
image and the limited color gamut digital image, both 
being encoded according to the storage color space. 

/5 The limited color gamut digital image 33 and the resid- 
ual image 36 are then stored in a digital tile 38 using a 
digital file storage step 37. 

[0037] Since most of the steps in this second method 
are identical to the corresponding steps in the first 

20 method, only the steps that differ will be discussed in 
more detail. The primary difference between the two 
methods is that the residual image is computed relative 
to the storage color space color values in this case. 
Therefore, the original extended color gamut digital 

25 image must be transformed to the storage color space 
in addition to the limited color gamut digital image. This 
is accomplished by the represent image in extended 
storage color space step 34. The complication is that 
the storage color space will typically only have a limited 

30 color gamut. For example, if the storage color space is a 
video RGB color space, then the color gamut of the stor- 
age space can be limited to the color gamut of the video 
display Therefore, to represent the original extended 
color gamut digital image in the storage color space, it is 

35 necessary to define an extended version of the storage 
color space that does not impose the limited color 
gamut. For example, 24-bit video RGB color spaces 
usually encode the color values in terms of integer code 
values in the range of 0 to 255. In order to allow the 

40 encoding of colors outside the color gamut of the video 
display the original extended color gamut digital image 
can be represented in an extended storage space 
where the code values were allowed to go outside the 
range 0 to 255. This would permit the encoding of colors 

45 with higher chroma values, as well as larger luminance 
dynamic range values, than can be encoded directly in 
the storage color space. After both the limited color 
gamut digital image and the extended color gamut dig- 
ital image had been represented in terms of the storage 

50 color space, the residual image 36 is then calculated as 
before by computing a difference between the two 
images. 

[0038] The result of applying the method of the 
present invention is the creation of both a limited color 
.5.5 gamut digital image in a storage color space and an 
associated residual image which correlates the limited 
color gamut digital image to an extended color gamut 
digital image. As discussed previously, the limited color 
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gamut digital image is generally well suited for display 
on a target output device such as a video display. One 
advantage of this approach is that systems that cannot 
make use of the residual image will be able to display 
and manipulate this image directly with no image quality 
or computation disadvantage relative to the prior art 
where only the limited color gamut digital image is 
stored. However, the information that normally would 
have been discarded has now been stored in the resid- 
ual image and is available for use by systems that can 
utilize it. In this case, the limited color gamut digital 
image Is extracted and the residual image from the dig- 
ital file is used to form a reconstructed extended color 
gamut digital image. 

[0039] FIG. 4 shows an example of reconstructing an 
extended color gamut digital image from the limited 
color gamut digital image and the residual image. The 
input to this process is an extended color gamut digital 
file 40 containing a limited color gamut digital image and 
a residual image created as described above. An 
extract data from digital file step 41 is used to extract the 
limited color gamut digital image 42 and the residual 
image 43. A reconstruct extended color gamut digital 
image step 44 is then used to form a reconstructed 
extended color gamut digital image 45 by combining the 
limited color gamut digital image 42 and the residual 
image 43. Typically the reconstruct extended color 
gamut digital image step 44 will involve combining the 
limited color gamut digital image 42 and the residual 
image 43. 

[0040] The reconstructed extended color gamut digital 
image can be used for many different purposes. For 
example, it can be used to form a digital image appropri- 
ate for display on an output device having a color gamut 
different from the limited color gamut of the limited color 
gamut digital image 42 in the digital file 40. This enables 
the generation of an optimal print from the original 
extended color gamut digital image, rather than a print 
limited by constraints of the storage color space. 
[0041] The information in the extended color gamut is 
particularly useful during the process of applying a 
modification to the digital image. Consider FIG. 5 which 
shows one method for modifying an extended color 
gamut digital image in accordance with the present 
invention. The input to this process is an extended color 
gamut digital file 40 containing a limited color gamut dig- 
ital image and a residual image created as described 
above. As in FIG. 4, an extract data from digital tile step 
41 is used to extract the limited color gamut digital 
image 42 and the residual image 43. One or more 
desired image modification (s) 50 are then specified to 
be used to modify the image. The limited color gamut 
digital image 42 is then modified using a modify limited 
color gamut digital image step 51 in response to the 
desired image modification (s) 50 to form a modified lim- 
ited color gamut digital image 53. Likewise, the residual 
image 43 is modified using a modify residual image step 
52 in response to the desired image modification(s) 50 



to form a modified residual image 54. The modified lim- 
ited color gamut digital image 53 and the modified resid- 
ual image 54 taken together represent a modified 
extended color gamut digital image which can be stored 

5 in a modified digital file 55. Alternatively, the modified 
digital image can be used to produce an image on a dig- 
ital printer, or some other form of image display device. 
The modified digital image can also be transmitted to a 
remote location for storage or printing at that location. 

10 Methods for transmitting the modified digital image can 
include the use of a computer network connection, or a 
modem connected to a telephone line, 
[0042] There are many different types of desired 
image modification (s) 50 that can be applied to the dig- 

is ital image. For example, there are many kinds of color 
and tone reproduction modifications that can be applied 
to an image. In some cases it will be desirable to adjust 
the color balance of the image. In other cases it will be 
desirable to adjust the density of the image so as to pro- 

20 duce a darker image or a lighter image. Other types of 
color and tone reproduction modifications can include 
changes in the contrast, hue or colorfulness of the 
image. In some cases, it will also be desirable to modify 
the color and tone reproductiorijCharacteristics to opti- 
cs mize them for a specific output device. The present 
invention is particularly well-suited to making modifica- 
tions to the color and tone reproduction characteristics 
of the image due to the fact that the extended color 
gamut information stored in the residual image will ena- 

30 ble larger changes to be made without any degradation 
in quality. For example, consider the case where an 
original image is determined to be over-exposed. In this 
case, highlight information in the limited color gamut 
digital image would have been clipped off during the 

35 process of rendering the image to the limited color 
gamut. However, this highlight information would be 
retained in the residual image. If it were desired to 
darken the image to correct for the over-exposure error, 
the information in the residual can then be used to 

40 determine a modified digital image that recovers the 
highlight detail. 

[0043] Other types of image modifications that can be 
applied to a digital image include spatial image modifi- 
cations. Examples of spatial image modifications 

45 include zooming, cropping, noise reduction, and sharp- 
ening the image. Zooming an image involves a resizing 
of the image. Cropping an image involves selecting a 
subset of an image. A noise reduction process generally 
involves reducing the appearance of image grain or 

50 noise by smoothing the image in flat areas of the image. 
Sharpening an image typically involves applying a spa- 
tial convolution to the image to increase the apparent 
sharpness of edges in the image. There are many other 
types of image modifications such as those found in 

55 common image editing software programs such as the 
widely used Adobe PhotoShop. 

[0044] In some cases a single image modification 
could be desired, whereas in other cases, it may be 
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desirable to apply several modifications to the image. 
For example, it might be desirable to darken the image, 
crop out a subset of the image, and then sharpen the 
image. Depending on the type of modifications that are 
desired, it is possible to apply the modification (s) simul- 
taneously, or it may be necessary to apply them 
sequentially. 

[0Q45] Modifications to the digital image can be inter- 
actively user specified, such as in the case of a user 
adjustable lightness knob. In this case, a preview of the 
image is generally displayed on a video display, and a 
user interface is provided to permit the user to specify 
his/her desired modification (s). Image modifications can 
also be determined by applying an automatic algorithm 
to the digital image. For example, a "scene balance 
algorithm" can be used to estimate the best color bal- 
ance and density level for an image. Alternatively an 
automatic sharpening algorithm can be used to esti- 
mate the optimal amount of sharpening for an image. 
[0046] In some cases, only the limited color gamut 
digital image may be available during the step of speci- 
fying the desirable image modification(s). For example, 
consider the case where the limited color gamut digital 
image is in a video RGB color space that is appropriate 
for display on a computer's video monitor. Certain soft- 
ware applications are designed to manipulate this video 
RGB image using an interactive user interface. The user 
interface can enable the user to control a color balance 
slidebar to determine the preferred color balance posi- 
tion. In this case, once the preferred color balance posi- 
tion were determined, this information can then be used 
to modify the extended color gamut digital image as 
described above. Similarly, the user can interactively 
determine desired zooming and cropping parameters 
based on the limited color gamut digital image and 
these changes can then be applied to the extended 
color gamut digital image. In some cases, even auto- 
matic algorithms that can be used to estimate the opti- 
mal image modification{s) can be designed to operate 
on a limited color gamut digital image. For example, a 
scene balance algorithm can be applied to a limited 
color gamut digital image, and the resulting image mod- 
ification(s) can then be applied to the extended color 
gamut digital image. Even if the limited color gamut dig- 
ital image is not stored in the color space used to pre- 
view the image, it may be necessary or convenient to 
provide a user interface where the desired image modi- 
fication (s) can be determined by manipulating a preview 
image which is determined from the limited color gamut 
digital image. 

[0047] Another embodiment of the present invention 
is shown in FIG. 6. In this case, a limited color gamut 
digital image 42 and a residual image 43 are used to 
form a reconstructed extended gamut digital image 45 
as was described above with reference to FIG. 4. 
Desired image modification{s) 61 are then used to form 
a modified extended color gamut digital image 62 using 
a modify reconstructed extended color gamut digital 
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image step 60. The modified extended color gamut dig- 
ital image 62 can then be used to determine a modified 
limited color gamut digital image 63 and a modified 
residual image 64 using the same methods that were 

5 described above. The modified limited color gamut dig- 
ital image 63 and the modified residual image 64 can 
then be optionally stored in a modified digital file 65. 
This method differs from that shown in FIG. 5 in that the 
desired image modification(s) are applied to the recon- 

w structed extended gamut digital image 60 rather than 
independently to the uncombined limited color gamut 
digital image 42 and residual image 43. In cases where 
complex image modification such as sharpening or 
color/tone adjustments are desired, it will be more con- 

J5 venient in many cases to use this approach. In other 
cases, such as for zoom and crop operations, it would 
not be necessary to go through the extra steps involved 
with forming the reconstructed image. 
[0048] Yet another embodiment of the present inven- 

20 tion is shown in FIG. 7. A limited color gamut digital 
image 42 and a residual image 43 are extracted from a 
digital file 40 using an extract data from digital file step 
41 as was describe in reference to FIG. 4. In this case, 
one or more desired image modification(s) 70 are 

25 applied to the limited color gamut digital image 42 using 
a modify limited color gamut digital image step 71 to 
form a modified limited color gamut digital image 72. An 
optional preview modified image step 73 will be useful in 
many cases so that a user can decide whether the opti- 
ma mal moditication(s) were applied to the image. Once the 
optimal desired image modification (s) 70 have been 
determined, information about the modification (s) 74 is 
then added to the input digital file 40 to form a modified 
digital file 75. For example, the information about the 

35 modification (s) 74 might include a list of the operations 
that were applied to the image, or a set of parameters 
describing the net effect of the modification (s). One 
case where this approach will be useful is when it is 
desired to preserve the original image in an unmodified 

40 form as long as possible. In another case of interest the 
digital file may exist at a remote site, and only a low-res- 
olution version of the limited color gamut digital image 
can be downloaded to a local workstation that is used to 
determine the image modification(s). It would only be 

45 necessary to send the information about the modifica- 
tion(s) 74 back to the remote site, rather than a modified 
high resolution image. In this case, it may be desirable 
to transmit a low-resolution version of the residual 
image to the local site as well for use in previewing the 

50 image modification (s). 

[0049] A computer program product having a compu- 
ter readable storage medium can have a computer pro- 
gram stored thereon for performing all the steps of the 
present invention. 

55 [0050] The computer readable storage medium can 
comprise, for example; magnetic storage media such as 
magnetic disc (such as a floppy disc) or magnetic tape; 
optical storage media such as optical disk, optical tape, 
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or machine readable bar code; solid state electronic 
storage devices such as random access memory 
(RAM), or read only memory (ROM); or any other phys- 
ical device or medium employed to store a computer 
program. 
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Claims 

1. A method for representing a digital image with an 
extended color gamut in a storage color space hav- 
5 ing a limited color gamut comprising the steps of: 

a) adjusting the color values of the extended 
color gamut digital image to fit within the limited 
color gamut to form a limited color gamut digital 

10 image; 

b) representing the limited color gamut digital 
image in the storage color space; 

c) determining one of more residual images 
representing differences between the extended 

15 color gamut digital image and the limited color 

gamut digital image; and 

d) associating the one of or more residual 
image(s) with the limited color gamut digital 
image in the storage color space such that the 

20 associated set of residual images and the lim- 

ited color gamut digital image is adapted to be 
used to form a reconstructed extended color 
gamut digital image. 

25 2. The method of claim 1 further including the step of 
using one or more residual images from the set of 
residual images together with the limited color 
gamut digital image to form a reconstructed 
extended color gamut digital image. 

30 

3. A method for representing a digital image having 
color values with an extended color gamut in a stor- 
age color space having a limited color gamut com- 
prising the steps of: 

35 

a) adjusting the color values of the extended 
color gamut digital image to fit within the limited 
color gamut to form a limited color gamut digital 
image; 

40 b) representing the limited color gamut digital 

image in the storage color space; 

c) compressing the limited color gamut digital 
image; 

d) determining a set of residual images repre- 
ss senting a difference between the extended 

color gamut digital image and an uncom- 
pressed version of the compressed limited 
color gamut digital image; and 

e) associating the set of residual images with 
50 the limited color gamut digital image in the stor- 
age color space such that the associated resid- 
ual images and the limited color gamut digital 
image is adapted to be used to form a recon- 
structed extended color gamut digital image. 

55 

4. A method for modifying a digital image having color 
values with an extended color gamut comprising 
the steps of: 
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a) representing the extended color gamut dig- 
ital image with a limited color gamut digital 
image and one or more associated residual 
image(s) representing a difference between 
the extended color gamut digital image and the 5 
limited color gamut digital image; 

b) specifying one or more desired image modi- 
tication(s) to be applied to the extended color 
gamut digital image; and 

c) modifying the limited color gamut digital 10 
image and the residual image(s) to form a mod- 
ified limited color gamut digital image and one 

or more modified residual image(s) in response 
to the specified desired image modification (s). 
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